Method, device and program for cutting out moving image

ABSTRACT

To enable obtainment of a composite image including a movement trajectory of a moving body, even from a moving image including a background. Sampling means samples frames from moving image data. First mask frame calculating means calculates mask frames including mask regions, which correspond to positions of the moving body on the frames. Second mask frame calculating means accumulates and binarizes the mask frames to calculate a reference mask frame. Region cutout means masks the frames by using the mask frames and the reference mask frame to cut out moving body regions, which correspond to a moving body, from the frames. Composing means overwrites moving body regions on one of the frames in chronological order to obtain a composite image, which includes the movement trajectory of the moving body.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to a device and a method forcutting out a moving image, which obtain a composite image including amovement trajectory of a moving body by composing a moving body includedin each of a plurality of frames sampled from the moving image on oneframe, and a program for causing a computer to execute the method forcutting out a moving image.

[0003] 2. Description of the Related Art

[0004] Due to the recent popularization of digital video cameras, it ispossible to handle a moving image on a per-frame basis. Moreover, acaptured moving image can be imported into a personal computer to beedited and processed as necessary. Furthermore, a method has beenproposed to process frames sampled from a moving image so that themovement trajectory of a moving object, in other words, a moving body,may appear in one still image (refer to Japanese Unexamined PatentPublication No. 5(1993)-316418)

[0005] In the method disclosed in Japanese Unexamined Patent PublicationNo. 5(1993)-316418, differential signals between a reference signal anda plurality of input signals are calculated. The calculated differentialsignals are added to the reference signal in chronological order togenerate composite signals. In the case where this composite signal isobtained, when the moving body overlaps with another, the signal valueexceeds an upper limit value of a video signal. In order to preventthis, the differential signals, of which values are halved, are added tothe reference signal.

[0006] Nevertheless, in the method, the signal used for the compositionincludes only the moving body without a background. Accordingly, whenthe method is applied to a plurality of frames, which are obtained byordinary filming and include both backgrounds and a moving body, themoving body cannot alone be extracted from the frames. As a result, thecomposite signals including the movement trajectory of the moving bodycannot be obtained. In addition, since the differential signals, ofwhich values are halved, are added to the reference signal in themethod, regions of the moving body overlap with the backgrounds in acomposite image obtained by reproducing the composite signals.Consequently, the contrast of the moving body is reduced.

SUMMARY OF THE INVENTION

[0007] The present invention has been devised in consideration of theforegoing circumstances. An object of the present invention is to beable to obtain a composite image including a movement trajectory of amoving body even from a moving image that includes a background.

[0008] A device for cutting out a moving image according to the presentinvention includes:

[0009] sampling means for sampling a plurality of sequential frames fromthe moving image including a moving body;

[0010] mask frame calculating means for calculating mask frames, whichinclude mask regions for identifying positions of the moving body in theplurality of frames, corresponding to each of the plurality of frames;and

[0011] region cutout means for cutting out moving body regionscorresponding to the moving body from all or part of the plurality offrames by masking all or part of the plurality of frames using the maskframes which correspond thereto.

[0012] In the device for cutting out a moving image according to thepresent invention,

[0013] the mask frame calculating means may be a means for calculatingother mask frames, which includes the mask regions for identifying thepositions of the moving body in other frames, corresponding to each ofthe other frames by calculating absolute values of difference values incorresponding pixels between a reference frame selected from theplurality of frames and the other frames and binarizing the absolutevalues based on a first threshold, and

[0014] the region cutout means may be a means for cutting out the movingbody regions corresponding to the moving body from the other frames bymasking the other frames using the other mask frames which correspondthereto.

[0015] In this case, the mask frame calculating means may filter theabsolute values of the difference values to remove noise and binarizethe absolute values which are filtered.

[0016] Moreover, in this case, the device for cutting out a moving imageaccording to the present invention may further include:

[0017] threshold correcting means for calculating a representative valueof pixel values indicating a level of the pixel values in each of theplurality of frames to correct the first threshold for each of theabsolute values of the difference values based on the representativevalue,

[0018] wherein the mask frame calculating means may be a means forbinarizing the absolute values based on the first threshold correctedfor each of the absolute values of the difference values.

[0019] In the device for cutting out a moving image according to thepresent invention,

[0020] the mask frame calculating means may include:

[0021] first mask frame calculating means for calculating other maskframes, which includes the mask regions for identifying the positions ofthe moving body in other frames, corresponding to each of the otherframes by calculating absolute values of difference values incorresponding pixels between a reference frame selected from theplurality of frames and the other frames and binarizing the absolutevalues based on a first threshold; and

[0022] second mask frame calculating means for calculating a referencemask frame, which includes a mask region for identifying a position ofthe moving body in the reference frame, based on information on theplurality of other mask frames, and

[0023] the region cutout means may be a means for cutting out a movingbody region corresponding to the moving body from the reference frame bymasking the reference frame using the reference mask frame.

[0024] Meanwhile, in the device for cutting out a moving image accordingto the present invention,

[0025] the mask frame calculating means may include:

[0026] first mask frame calculating means for calculating other maskframes, which includes the mask regions for identifying the positions ofthe moving body in other frames, corresponding to each of the otherframes by calculating absolute values of difference values incorresponding pixels between a reference frame selected from theplurality of frames and the other frames and binarizing the absolutevalues based on a first threshold; and

[0027] second mask frame calculating means for calculating a referencemask frame, which includes a mask region for identifying a position ofthe moving body in the reference frame, based on information on theplurality of other mask frames, and

[0028] the region cutout means may be a means for cutting out the movingbody regions corresponding to the moving body from the plurality offrames by masking the other frames using the other mask frames whichcorrespond thereto and masking the reference frame using the referencemask frame.

[0029] In this case, the firstmask frame calculating means may be ameans for filtering the absolute values of the difference values toremove noise and binarizing the absolute values which have beenfiltered.

[0030] Various kinds of filters can be used for the filtering, as longas the filters remove noise. Examples of the filters are a low passfilter, a median filter, a maximum value filter, and a minimum valuefilter.

[0031] Moreover, in this case, the device for cutting out a moving imageaccording to the present invention may further include:

[0032] threshold correcting means for calculating a representative valueof pixel values indicating a level of the pixel values in each of theplurality of frames to correct the first threshold for each of theabsolute values of the difference values based on the representativevalue,

[0033] wherein the first mask frame calculating means is a means forbinarizing the absolute values based on the first threshold correctedfor each of the absolute values of the difference values.

[0034] When the device for cutting out a moving image according topresent invention include the second mask frame calculating means, thesecond mask frame calculating means may be a means for calculating thereference mask frame by accumulating or weighting and accumulatingcorresponding pixels in a selection mask frame selected from theplurality of other mask frames and further binarizing the accumulatedselection mask frame based on a second threshold.

[0035] There are cases in which the moving body regions cannot be cutout when the moving body moves slowly and the mask regions becomeextremely small in the other mask frames and the reference mask frame.Thus, when the device for cutting out a moving image according to thepresent invention is provided with the second mask frame calculatingmeans, the device may further include selecting means for comparing asize of the mask region in each of the other mask frames with a thirdthreshold to select the other mask frame with the mask region having thesize exceeding the third threshold as the selection mask frame.

[0036] In addition, the second mask frame calculating means may be ameans for filtering the accumulated selection mask frame to remove noiseand binarizing the accumulated selection mask frame which has beenfiltered.

[0037] Various kinds of filters can be used for the filtering, as longas the filters remove noise. Examples of the filters are a low passfilter, a median filter, a maximum value filter, and a minimum valuefilter.

[0038] Note that there are cases in which the moving body regions cannotbe cut out by using the mask frames and the reference mask frame, whenthe sampling means samples only two frames or when the moving body movesslowly and there are numerous regions in the accumulated selection maskframe where the moving bodies overlap.

[0039] Accordingly, when the device for cutting out a moving imageaccording to the present invention includes the second mask framecalculating means, the second mask frame calculating means may be ameans for re-calculating the reference mask frame by calculating colorinformation on a region in the reference frame, which corresponds to themask region in the reference mask frame; weighting the region in theaccumulated selection mask frame, which corresponds to a region havingcolor information similar to the color information; and binarizing theweighted and accumulated selection mask frame based on the secondthreshold.

[0040] Moreover, the device for cutting out a moving image according tothe present invention may further include:

[0041] inputting means for accepting inputs of a first number of frames,a second number of frames, and a third number of frames, the firstnumber of frames indicating a number of the other frames having a regionwhich corresponds to the moving body and does not overlap with a regioncorresponding to the moving body in the reference frame, the secondnumber of frame indicating a number of the other frames having a regionwhich corresponds to the moving body and overlaps with the regioncorresponding to the moving body in the reference frame, and the thirdnumber of frames indicating a number of the other frames having regionswhich correspond to the moving body and overlap with each other, in thecase that the plurality of frames are overlapped; and

[0042] judging means for judging whether a condition the first number offrames>the second number of frames>the third number of frames existsamong the first number of frames, the second number of frames, and thethird number of frames, which are inputted, and for allowing the regioncutout means to cut out the moving body region only when the conditionis satisfied.

[0043] The device for cutting out a moving image according to thepresent invention may further include:

[0044] aligning means for aligning background regions other than themoving body region in the plurality of frames,

[0045] wherein the mask frame calculating means may be a means forcalculating the mask frame from the plurality of frames after alignment.

[0046] The device for cutting out a moving image according to thepresent invention may further include:

[0047] frame correcting means for correcting pixel values of theplurality of frames so that each representative value, such as a meanvalue or a median value of the pixel values or a maximum frequency valueof the histogram, may substantially match another by calculating therepresentative value of the pixel values, which indicates a level of thepixel values for each of the plurality of frames,

[0048] wherein the mask frame calculating means may be a means forcalculating the mask frames from the plurality of frames after beingcorrected.

[0049] The device for cutting out a moving image according to thepresent invention may further include composing means for obtaining acomposite image of the plurality of frames by overwriting the movingbody regions, which are cut out from the plurality of frames by theregion cutout means, in chronological order at positions correspondingto the moving body regions in one of the plurality of frames.

[0050] A method for cutting out a moving image includes the steps of:

[0051] sampling a plurality of sequential frames from the moving imageincluding a moving body;

[0052] calculating mask frames, which include mask regions foridentifying positions of the moving body in the plurality of frames,corresponding to each of the plurality of frames; and

[0053] cutting out moving body regions corresponding to the moving bodyfrom all or part of the plurality of frames by masking all or part ofthe plurality of frames using the mask frames which correspond thereto.

[0054] Note that the method for cutting out a moving image according tothe present invention may be provided as a program for causing acomputer to execute the method.

[0055] According to the present invention, the sampling means samplesthe plurality of sequential frames from the moving image. The mask framecalculating means calculates the mask frames, which include the maskregions for identifying the positions of the moving body in theplurality of frames, corresponding to each of the plurality of frames.Herein, the mask regions correspond to the moving body regions in theframes. The region cutout means masks all or part of the plurality offrames by using the mask frames, which correspond thereto, to cut outthe moving body regions corresponding to the moving body from the all orpart of the plurality of frames.

[0056] Accordingly, it is possible to cut out the moving body regionsfrom the frames appropriately even when the moving image includes boththe background and the moving body. Thus, the composite image includingthe movement trajectory of the moving body can be easily obtained byusing the cutout moving body regions.

[0057] The mask frame calculating means and the region cutout meansperform relatively simple processes. The mask frame calculating meanscalculates the mask frames, which include the mask regions, of theplurality of frames. The region cutout means performs the masking.Consequently, it is possible to cut out the moving body regions easilyfrom the frames.

[0058] In the device for cutting out a moving image according to thepresent invention, the mask frame calculating means and the regioncutout means may perform the processes as in the second aspect of thepresent invention. The mask frame calculating means may calculate theother mask frames, which include the mask regions for identifying thepositions of the moving body in the other frames, corresponding to eachof the other frames by calculating the absolute values of the differencevalues in the corresponding pixels between the reference frame selectedfrom the plurality of frames and the other frames and binarizing theabsolute values based on the first threshold. The region cutout meansmay cut out the moving body regions corresponding to the moving bodyfrom the other frames by masking the other frames using the other maskframes which correspond thereto.

[0059] Accordingly, it is possible to cut out the moving body regionsfrom the other frames appropriately even when the moving image includesboth the background and the moving body. Thus, the composite imageincluding the movement trajectory of the moving body can be easilyobtained by using the cutout moving body regions.

[0060] The mask frame calculating means and the region cutout meansperform relatively simple processes. The mask frame calculating meanscalculates and binarizes the absolute values of the difference values.The region cutout means performs the masking. Consequently, it ispossible to cut out the moving body regions easily from the otherframes.

[0061] According to the device for cutting out a moving image of thethird aspect of the present invention, the mask frame calculating meansfilters the absolute values of the difference values to remove noise andbinarizes the filtered absolute values. Herein, the regions havingrelatively large absolute values originally correspond to the movingbody included in the reference frame and the other frames. However, eachframe includes noise upon filming and noise due to the movement of themoving body, and thus small regions having small absolute values areoccasionally included in the regions having relatively large absolutevalues. On the contrary, small regions having relatively large absolutevalues are occasionally included in the regions having small absolutevalues. Thus, by filtering the absolute values of the difference valuesto remove noise, it is possible to remove noise due to these smallregions from the absolute values. Therefore, the other mask frames canbe precisely calculated while the effects of the noise are reduced.

[0062] In the device for cutting out a moving image according to thepresent invention, the first mask frame calculating means, the secondmask frame calculating means, and the region cutout means may performthe processes as in the fifth aspect of the present invention. The firstmask frame calculating means may calculate the other mask frames, whichinclude the mask regions for identifying the positions of the movingbody in the other frames, corresponding to each of the other frames bycalculating the absolute values of the difference values in thecorresponding pixels between the reference frame selected from theplurality of frames and the other frames and binarizing the absolutevalues based on the first threshold. The second mask frame calculatingmeans may calculate the reference mask frame based on the information onthe plurality of other frames. The region cutout means may cut out themoving body region corresponding to the moving body from the referenceframe by masking the reference frame using the reference mask frame.

[0063] Accordingly, it is possible to cut out the moving body regionfrom the reference frame appropriately even when the moving imageincludes both the background and the moving body. Thus, the compositeimage including the movement trajectory of the moving body can be easilyobtained by using the cutout moving body region.

[0064] The first mask frame calculating means, the second mask framecalculating means, and the region cutout means perform relatively simpleprocesses. The first mask frame calculating means calculates andbinarizes the absolute values of the difference values. The second maskframe calculating means performs the accumulation and the binarization.The region cutout means performs the masking. Consequently, it ispossible to cut out the moving body region easily from the referenceframe.

[0065] In the device for cutting out a moving image according to thepresent invention, the mask frame calculating means may include thefirst mask frame calculating means and the second mask frame calculatingmeans as in the sixth aspect of the present invention. The first maskframe calculating means calculates the other mask frames, which includethe mask regions for identifying the positions of the moving body in theother frames, corresponding to each of the other frames by calculatingthe absolute values of the difference values in the corresponding pixelsbetween the reference frame selected from the plurality of frames andthe other frames and binarizing the absolute values based on the firstthreshold. The second mask frame calculating means calculates thereference mask frame, which includes the mask region for identifying theposition of the moving body in the reference frame, based on theinformation on the plurality of mask frames. This makes it possible tocalculate the other mask frames by simple processes such as thecalculation and binarization of the absolute values of the differencevalues.

[0066] According to the device for cutting out a moving image of theseventh aspect of the present invention, the first mask framecalculating means filters the absolute values of the difference valuesto remove noise and binarizes the filtered absolute values. Herein, theregions having relatively large absolute values originally correspond tothe moving body included in the reference frame and the other frames.However, each frame includes noise upon filming and noise due to themovement of the moving body, and thus small regions having smallabsolute values are occasionally included in the regions havingrelatively large absolute values. On the contrary, small regions havingrelatively large absolute values are occasionally included in theregions having small absolute values. Thus, by filtering the absolutevalues of the difference values to remove noise, it is possible toremove noise due to these small regions from the absolute values.Therefore, the other mask frames can be precisely calculated while theeffects of the noise are reduced.

[0067] According to the device for cutting out a moving image of thefourth and eighth aspects of the present invention, the thresholdcorrecting means calculates a representative value of pixel valuesindicating a level of the pixel values for each of the plurality offrames and corrects the first threshold for each of the absolute valuesof the difference values based on the representative value. The maskframe calculating means or the first mask frame calculating meansbinarizes the absolute values based on the first threshold corrected foreach of the absolute values of the difference values. Herein, when thecontrast adjusting function of a video camera used for filming themoving image does not work well or when it becomes suddenly cloudyduring outdoor filming in fine weather, the entire contrast differsbetween the sampled frames. Thus, when the mask frame calculating meansor the first mask frame calculating means calculates the absolute valuesof the difference values between the corresponding pixels in thereference frame and the other frames by using such frames, the absolutevalues will become relatively large even in the background regions whichdo not correspond to the moving body. Therefore, it is impossible toaccurately calculate the mask frames having mask regions whichcorrespond to only the moving body by the binarization.

[0068] Thus, the first threshold is corrected for each of the absolutevalues of the difference values to perform the binarization as in thedevice for cutting out a moving image of the fourth and eighth aspectsof the present invention. This makes it possible to appropriatelycalculate the mask frames having the mask regions which correspond toonly the moving body, even when the absolute values are relativelylarge.

[0069] When the device for cutting out a moving image is provided withthe second mask frame calculating means, the second mask framecalculating means calculates the reference mask frame by accumulatingthe corresponding pixels in the selection mask frame selected from theplurality of other frames and binarizing the accumulated selection maskframe based on the second threshold. This makes it possible to calculatethe reference mask frame by the simple processes including theaccumulation and the binarization.

[0070] There are cases in which the mask region in the reference maskframe does not correspond to the moving body region included in theframe when the moving body moves slowly and the mask regions becomeextremely small in the other mask frames. Hence, as in the device forcutting out a moving image of the tenth aspect of the present invention,the selecting means compares the size of the mask region in each of theother mask frames with the third threshold and selects the other maskframe having the mask region with a size exceeding the third thresholdas the selection mask frame. This makes it possible to calculate thereference mask frame including the mask region which corresponds to themoving body region.

[0071] According to the device for cutting out a moving image of theeleventh aspect of the present invention, the second mask framecalculating means filters the accumulated selection mask frame to removenoise and binarizes the filtered selection mask frame. Herein, theregions having relatively large absolute values in the accumulatedselection mask frames originally correspond to the moving body includedin the reference frame and the other frames. However, each frameincludes noise upon filming and noise due to the movement of the movingbody, and thus small regions having small values in the accumulatedselection mask frame are occasionally included in the regions havingrelatively large absolute values in the accumulated selection maskframe. On the contrary, small regions having relatively large values inthe accumulated selection mask frame are occasionally included in theregions having small values in the accumulated selection mask frame.Thus, by filtering the accumulated selection mask frame to remove noise,it is possible to remove noise due to these small regions from theaccumulated selection mask frame. Therefore, the reference mask framecan be precisely calculated while the effects of the noise are reduced.

[0072] As in the twelfth aspect of the present invention, the colorinformation on the region in the reference frame, which corresponds tothe mask region in the reference mask frame, is calculated. The regionin the accumulated selection mask frame, which corresponds to the regionhaving the color information similar to the color information, isweighted. The weighted and accumulated selection mask frame is binarizedbased on the second threshold to re-calculate the reference mask frame.Therefore, it is possible to ensure that the moving body regioncorresponding to the moving body is cut out from the reference frame.

[0073] Assume a case in which the plurality of frames are overlapped.The first number of frames is defined as the number of other frameshaving regions which correspond to a moving body and do not overlap withthe region corresponding to the moving body in the reference frame. Thesecond number of frames is defined as the number of other frames havingregions which correspond to the moving body and overlap with the regionscorresponding to the moving body in the reference frame. The thirdnumber of frames is defined as the number of other frames having regionswhich correspond to the moving body and overlap with each other. Onlywhen the condition the first number of frames>the second number offrames>the third number of frames is satisfied, it is possible to cutout the moving body regions from all the frames. Thus, as in the devicefor cutting out the moving image according to the thirteenth aspect ofthe present invention, if the inputting means accepts the inputs of thefirst to third numbers of frames, the judging means judges whether thecondition the first number of frames>the second number of frames>thethird number of frames is satisfied, and the region cutout means cutsout the moving body regions only when the condition is satisfied, it ispossible to ensure that the moving body regions corresponding to themoving body are cut out from the frames.

[0074] According to the device for cutting out a moving image of thefourteenth aspect of the present invention, the aligning means alignsthe background regions in the plurality of frames and the mask framecalculating means calculates the mask frames from the plurality ofaligned frames. Accordingly, it is possible to remove the effects of themovement of the backgrounds, due to wobbling or the like upon filming,from the frames. Therefore, the moving body regions can be accuratelycut out from the frames while the effects of the movement of thebackground regions are eliminated.

[0075] According to the device for cutting out a moving image of thefifteenth aspect of the present invention, the frame correcting meanscalculates a representative value of pixel values indicating a level ofthe pixel values for each of the plurality of frames and corrects thepixel values of the plurality of frames to substantially correspond tothe representative value. The mask frame calculating means calculatesthe mask frames from the corrected frames. Herein, when the contrastadjusting function of a video camera used for filming the moving imagedoes not work well or when it becomes suddenly cloudy during outdoorfilming in fine weather, the entire contrast differs between the sampledframes. Thus, when the mask frame calculating means calculates theabsolute values of the difference values in the corresponding pixelsbetween the reference frame and the other frames by using such frames,the absolute values will become relatively large even in the backgroundregions which do not correspond to the moving body. Therefore, it isimpossible to accurately calculate the mask frames having mask regionswhich correspond to only the moving body.

[0076] Consequently, as in the device for cutting out a moving image ofthe fifteenth aspect of the present invention, by correcting the pixelvalues of the plurality of frames, the entire contrast in each frame canbe substantially the same. As a result, the absolute values will not beextremely large in the background regions, which do not correspond tothe moving body. Therefore, it is possible to accurately calculate themask frames having mask regions which correspond to only the movingbody.

[0077] According to the device for cutting out a moving image of thesixteenth aspect of the present invention, the composing meansoverwrites the moving body regions at the positions in chronologicalorder. These moving body regions correspond to all or part of theplurality of frames cut out by the region cutout means, and thepositions in one of the plurality of frames correspond to the movingbody-regions. Thus, a composite image including the movement trajectoryof the moving body can be obtained without reducing the contrast betweenthe moving body and the background.

BRIEF DESCRIPTION OF THE DRAWINGS

[0078]FIG. 1 is a schematic block diagram showing the configuration of asystem for composing a moving image according to a first embodiment ofthe present invention, which includes a device for cutting out a movingimage.

[0079]FIG. 2 is a diagram showing examples of sampled frames.

[0080]FIG. 3 is a diagram showing mask frames.

[0081]FIG. 4 is a diagram showing a cumulative frame.

[0082]FIG. 5 is a diagram showing a reference mask frame.

[0083]FIG. 6 is a flow chart showing the process performed in the firstembodiment.

[0084]FIG. 7 is a diagram for explaining the process performed in thefirst embodiment by using frames.

[0085]FIG. 8 is a schematic block diagram showing the configuration of asystem for composing a moving image according to a second embodiment ofthe present invention, which includes a device for cutting out a movingimage.

[0086]FIG. 9 is a diagram showing an example of a cumulative frameobtained from six frames.

[0087]FIG. 10A is a diagram showing an example of a cumulative frameobtained from three frames.

[0088]FIG. 10B is a diagram showing an example of a cumulative frameobtained from two frames.

[0089]FIG. 11 is a schematic block diagram showing the configuration ofa system for composing a moving image according to a third embodiment ofthe present invention, which includes a device for cutting out a movingimage.

[0090]FIG. 12 is a diagram showing examples of sampled frames.

[0091]FIG. 13 is a diagram showing mask frames.

[0092]FIG. 14A is a diagram showing a cumulative frame.

[0093]FIG. 14B is a diagram showing a reference mask frame.

[0094]FIG. 15 is a diagram for explaining the calculation of a compositeimage.

[0095]FIG. 16A is a diagram showing a cumulative frame.

[0096]FIG. 16B is a diagram showing a reference mask frame.

[0097]FIG. 17 is a diagram for explaining the calculation of a compositeimage.

[0098]FIG. 18 is a schematic block diagram showing the configuration ofa system for composing a moving image according to a fourth embodimentof the present invention, which includes a device for cutting out amoving image.

[0099]FIG. 19 is a diagram showing a color region obtained in the fourthembodiment.

[0100]FIG. 20A is a diagram showing a cumulative frame.

[0101]FIG. 20B is a diagram showing a reference mask frame.

[0102]FIG. 21 is a schematic block diagram showing the configuration ofa system for composing a moving image according to a fifth embodiment ofthe present invention, which includes a device for cutting out a movingimage.

[0103]FIG. 22 is a diagram for explaining the alignment by aligningmeans.

[0104]FIG. 23 is a diagram showing an example of a filter used byfiltering means.

[0105]FIG. 24 is a flow chart showing the process performed in the fifthembodiment.

[0106]FIG. 25 is a schematic block diagram showing the configuration ofa system for composing a moving image according to a sixth embodiment ofthe present invention, which includes a device for cutting out a movingimage.

[0107]FIG. 26 is a schematic block diagram showing the configuration ofa system for composing a moving image according to a seventh embodimentof the present invention, which includes a device for cutting out amoving image.

[0108]FIG. 27A is a diagram showing a difference frame.

[0109]FIG. 27B is a diagram showing the difference frame after opening.

[0110]FIG. 28 is a diagram for explaining the alignment by aligningmeans in an eighth embodiment.

[0111]FIG. 29 is a diagram for explaining the process performed in aninth embodiment by using frames.

[0112]FIG. 30 is a schematic block diagram showing the configuration ofa system for composing a moving image according to a tenth embodiment ofthe present invention, which includes a device for cutting out a movingimage.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0113] Hereinafter, embodiments of the present invention are describedwith reference to the drawings. FIG. 1 is a schematic block diagramshowing the configuration of a system for composing a moving imageaccording to a first embodiment of the present invention, which includesa device for cutting out a moving image. As shown in FIG. 1, the systemfor composing a moving image according to the first embodiment of thepresent invention comprises sampling means 1, first mask framecalculating means 2, second mask frame calculating means 3, regioncutout means 4, and composing means 5.

[0114] The sampling means 1 samples a plurality of frames Fri (i=1 to N)from inputted moving image data D0. In the first embodiment, four framesFr1 to Fr4 are sampled from the moving image data D0 as shown in FIG. 2.The moving image data D0 represents a moving image, in which a circularmoving body X0 is moving. Note that the value of i increases for theframe Fri as time elapses.

[0115] The first mask frame calculating means 2 comprises subtractingmeans 2A and binarizing means 2B.

[0116] The subtracting means 2A calculates absolute values |di(x, y)|(i=1 to 3) of difference values in corresponding pixels between areference frame (in the first embodiment, the frame Fr4), which isselected from the plurality of frames Fr1 to Fr4, and other frames Fr1to Fr3 by using the following equation (1):

|di(x, y)|=|Fri(x, y)−Fr 4(x, y)|  (1)

[0117] where (x, y) are coordinates of corresponding pixels between theframes Fri and Fr4.

[0118] Color data RGB or luminance and chrominance data YCC form themoving image data D0 and the frames Fr1 to Fr4. Accordingly, absolutevalues of difference values between the reference frame Fr4 and theother frames Fr1 to Fr3 are calculated in terms of each component of RGBor YCC. After each component is weighted, the Euclidean distance isobtained for each component and may be used as an absolute value of thedifference value. For example, when the luminance and chrominance dataYCC form the frames Fr1 to Fr4, the absolute value |di (x, y)| of thedifference value in each pixel position is calculated by using thefollowing equation (2):

|di(x,y)|={square root}{square root over ((|a·Yd(x,y)|² +|b·Cbd(x,y)|²+|c·Crd(x,y)|²)}  (2)

[0119] where Yd is an absolute value of a difference value calculatedfor luminance Y, Cbd is an absolute value of a difference valuecalculated for chrominance Cb, and Crd is an absolute value of adifference value calculated for chrominance Cr.

[0120] The binarizing means 2B binarizes the absolute value |di(x, y)|of the difference value based on a threshold Th1. Specifically, a valueof 1 is given to a pixel having the absolute value |di(x, y)| of adifference value larger than the threshold Th1. A value of 0 is given toa pixel having the absolute value less than or equal to the thresholdTh1. Mask frames M1 to M3, which include pixels with a value of 1 asmask regions and correspond to the other frames Fr1 to Fr3, arecalculated. The mask frames M1 to M3 thus calculated are shown in FIG.3. Note that shaded areas indicate the mask regions in FIG. 3. The maskframe M1 has mask regions R1A and R1B. The mask frame M2 has maskregions R2A and R2B. The mask frame M3 has mask regions R3A and R3B.Herein, the mask regions R1A, R2A and R3A of the mask frames M1 to M3correspond to regions of the moving body X0 in the frames Fr1 to Fr3,respectively. The mask regions R1B, R2B, and R3B correspond to a regionof the moving body X0 in the reference frame Fr4.

[0121] The second mask frame calculating means 3 comprises accumulatingmeans 3A and binarizing means 3B.

[0122] The accumulating means 3A accumulates the corresponding pixels inall the mask frames M1 to M3 to calculate a cumulative frame Frc byusing the equation (3) below. Note that all the mask frames M1 to M3 areselected for the calculation of the undermentioned reference mask frameFrc1 in the first embodiment.

Frc(x,y)=ΣMi(x, y)(i=1 to 3)   (3)

[0123] Note that the mask frames M1 to M3 may be weighted andaccumulated to calculate the reference mask frame Frc1.

[0124]FIG. 4 is a diagram showing the cumulative frame Frc. As shown inFIG. 4, in the cumulative frame Frc, a region, which corresponds to onlythe mask region R1A of the mask frame M1, has a value of 1. A region,which corresponds to only the mask region R2A of the mask frame M2, hasa value of 1. A region, which corresponds to only the mask region R3A ofthe mask frame M3, has a value of 1. A region, which corresponds to boththe mask region R1B of the mask frame M1 and the mask region R2B of themask frame M2 (i.e., a region having a value of 0, where the maskregions R3A and R3B overlap in the mask frame M3), has a value of 2. Aregion, which corresponds to the mask region R1B of the mask frame M1,the mask region R2B of the mask frame M2, and the mask region R3B of themask frame M3, has a value of 3.

[0125] The binarizing means 3B binarizes the cumulative frame Frc basedon a threshold Th2. Herein, for example, the threshold Th2 is set to avalue of 1 (when the total number of frames-k, k=3). When a pixel valuein the cumulative frame Frc exceeds the threshold Th2, a value of 1 isgiven to the pixel. When a pixel value is less than or equal to thethreshold Th2, a value of 0 is given to the pixel. Thereafter, areference mask frame Frc1 is calculated, in which a pixel having a valueof 1 is set to be a mask region. This calculated reference mask frameFrc1 is shown in FIG. 5. Note that a shaded area, a region R4A,represents a mask region in FIG. 5. Note that the mask region in thereference mask frame Frc1 corresponds to the region of the moving bodyX0 in the reference frame Fr4.

[0126] The region cutout means 4 masks the frames Fr1 to Fr3 by usingthe corresponding mask frames M1 to M3. The region cutout means 4 alsomasks the reference frame Fr4 by using the reference mask frame Frc1.Accordingly, the region cutout means 4 cuts out a moving body regioncorresponding to the moving body X0 from each of the plurality of framesFr1 to Fr4. Specifically, as for the frame Fr1, regions, whichcorrespond to the mask regions R1A and R1B having the pixel values of 1in the mask frame M1, are cut out as moving body regions A1A and A1B(hereinafter, generically referred to as A1 occasionally). As for theframe Fr2, regions, which correspond to the mask regions R2A and R2Bhaving the values of 1 in the mask frame M2, are cut out as moving bodyregions A2A and A2B (hereinafter, generically referred to as A2occasionally). As for the frame Fr3, regions, which correspond to themask regions R3A and R3B having the values of 1 in the mask frame M3,are cut out as moving body regions A3A and A3B (hereinafter, genericallyreferred to as A3 occasionally).

[0127] Note that the moving body regions A1B, A2B, and A3B correspond tobackground regions in the frames Fr1 to Fr3, instead of the regions ofthe moving body X0. Thus, the cutout moving body regions A1B, A2B, andA3B have substantially the same image.

[0128] Meanwhile, as for the reference frame Fr4, a region, whichcorresponds to the mask region R4A having the value of 1 in thereference mask frame Frc1, is cut out as a moving body region A4A(hereinafter, referred to as A4 occasionally).

[0129] The composing means 5 overwrites the moving body regions A1 to A4in chronological order at positions corresponding to the moving bodyregions A1 to A4 in one of the frames Fr1 to Fr4 (herein, Fr1). Thus,the composing means 5 obtains a composite image G0 of the plurality offrames Fr1 to Fr4.

[0130] Next, the process performed in the first embodiment is described.FIG. 6 is a flow chart showing the process performed in the firstembodiment. FIG. 7 is a diagram for explaining the process performed inthe first embodiment by using the frames. First, the sampling means 1samples the four frames Fr1 to Fr4 from the moving image data D0 (StepS1). Second, the subtracting means 2A of the first mask framecalculating means 2 calculates the absolute values |di(x, y)| of thedifference values in the corresponding pixels between the referenceframe Fr4 and the other frames Fr1 to Fr3 (Step S2). Third, thebinarizing means 2B binarizes the absolute values of the differencevalues based on the threshold Th1 to calculate the mask frames M1 to M3(Step S3).

[0131] Fourth, the accumulating means 3A of the second mask framecalculating means 3 accumulates the corresponding pixels in all the maskframes M1 to M3 to calculate the cumulative frame Frc (Step S4). Fifth,the binarizing means 3B binarizes the cumulative frame Frc based on thethreshold Th2 to calculate the reference mask frame Frc1 (Step S5).

[0132] Sixth, the region cutout means 4 masks the frames Fr1 to Fr3 byusing the corresponding mask frames M1 to M3. The region cutout means 4also masks the reference frame Fr4 by using the reference mask frameFrc1. Accordingly, the region cutout means 4 cuts out the moving bodyregions A1 to A4, which correspond to the moving body X0, from theplurality of frames Fr1 to Fr4 (Step S6).

[0133] Seventh, the composing means 5 overwrites the moving body regionsA1 to A4 in chronological order at the positions corresponding to themoving body regions A1 to A4 in the frame Fr1, one of the frames Fr1 toFr4. Thus, the composing means 5 obtains the composite image G0 of theplurality of frames Fr1 to Fr4 (Step S7). Hence, the process iscompleted.

[0134] Specifically, a first composite image GFr1 is obtained byoverwriting the moving body regions A1A and A1B, which are cut out fromthe frame Fr1, on the frame Fr1, as shown in the lowermost row in FIG.7. In the first embodiment, since the moving body regions A1 to A4 areoverwritten on the frame Fr1, overwriting the moving body regions A1Aand A1B onto the frame Fr1 can be omitted. Note that the moving bodyregion A1A corresponds to the region of the moving body X0 in the frameFr1, and the moving body region A1B corresponds to the region of themoving body X0 in the frame Fr4.

[0135] Next, a second composite image GFr1+2 is obtained by overwritingthe moving body regions A2A and A2B, which are cut out from the frameFr2, on the first composite image GFr1. Note that the moving body regionA2A corresponds to the region of moving body X0 in the frame Fr2, andthe moving body region A2B corresponds to the region of the moving bodyX0 in the frame Fr4. Consequently, the moving body region A2B isoverwritten on the moving body region A1B in the composite image GFr1.However, since the moving body regions A1B and A2B representapproximately the same image, the image in the moving body region A1Bwill not be substantially changed in the composite image GFr1.

[0136] Subsequently, a third composite image GFr1+2+3 is obtained byoverwriting the moving body regions A3A and A3B, which are cut out fromthe frame Fr3, on the second composite image GFr1+2. Note that themoving body region A3A corresponds to the region of the moving body X0in the frame Fr3, and the moving body region A3B corresponds to theregion of the moving body X0 in the frame Fr4. Accordingly, the movingbody region A3B is overwritten on the moving body region A2B partiallyin the composite image GFr1+2. However, the moving body regions A2B andA3B represent substantially the same image. Thus, the image in themoving body region A2B will not be substantially changed in thecomposite image GFr1+2.

[0137] At last, a fourth composite image GFr1+2+3+4, in other words, afinal composite image G0 is obtained by overwriting the moving bodyregion A4A, which is cut out from the frame Fr4, on the third compositeimage GFr1+2+3. Note that the moving body region A4A corresponds to theregion of the moving body X0 in the frame Fr4. Accordingly, the movingbody region A4A is overwritten on the region corresponding to the movingbody region A3B and a fragment of the circle of the moving body regionA3A in the composite image GFr1+2+3. Thus, the movement trajectory ofthe moving body X0 appears on the composite image G0.

[0138] As described above, the mask regions of the mask frames M1 to M3,which are calculated in the first embodiment, represent the regions ofthe moving body X0 included in the frames Fr1 to Fr3. The reference maskframe Frc1 represents the region of the moving body X0 included in thereference frame Fr4. As a result, the mask frames M1 to M3 and thereference mask frame Frc1 are used to mask the frames Fr1 to Fr3 and thereference frame Fr4, respectively. This makes it possible to cut out themoving body regions A1 to A4 from the frames Fr1 to Fr4 appropriatelyeven when the moving image data D0 includes both backgrounds and themoving body X0. Therefore, the composite image including the movementtrajectory of the moving body X0 can be easily obtained by using thecutout moving body regions A1 to A4.

[0139] The first mask frame calculating means 2, the second mask framecalculating means 3, and the region cutout means 4 perform relativelysimple processes. The first mask frame calculating means 2 calculatesand binarizes the absolute values of the difference values. The secondmask frame calculating means 3 performs accumulation and binarization.The region cutout means 4 performs masking. Therefore, it is possible tocut out the moving body regions A1 to A4 easily from the frames Fr1 toFr4.

[0140] Next, a second embodiment of the present invention is described.FIG. 8 is a schematic block diagram showing the configuration of asystem for composing a moving image according to a second embodiment ofthe present invention, which includes a device for cutting out a movingimage. Note that the components in the second embodiment, which are thesame as those in the first embodiment, are given the same referencenumerals, and details thereof are omitted.

[0141] The system for composing a moving image according to the secondembodiment further comprises inputting means 6, judging means 7, and amonitor 8, in addition to the components in the first embodiment. Theinputting means 6 accepts inputs of the first number of frames C1, thesecond number of frames C2, and the third number of frames C3. In thecase where a plurality of frames Fr1 to Fr4 are overlapped, the firstnumber of frames C1 indicates the number of other frames having regionswhich correspond to a moving body X0 and do not overlap with a regioncorresponding to the moving body X0 in a reference frame Fr4. The secondnumber of frames C2 indicates the number of other frames having regionswhich correspond to the moving body X0 and overlap with the regioncorresponding to the moving body X0 in the reference frame Fr4. Thethird number of frames C3 indicates the number of other frames havingregions which correspond to the moving body X0 and overlap with eachother. The judging means 7 judges whether the condition C1>C2>C3 issatisfied among the inputted first number of frames C1, second number offrames C2, and third number of frames C3. Only when the condition issatisfied, the judging means 7 allows region cutout means 4 to cut outmoving body regions. The monitor 8 displays a cumulative frame Frcobtained when a reference mask frame Frc1 is obtained.

[0142] The inputting means 6 accepts the first number of frames C1, thesecond number of frames C2, and the third number of frames C3, which areinputted by an operator manipulating the system for composing a movingimage according to the second embodiment. Those numbers are obtained bythe operator monitoring a cumulative frame Frc displayed on the monitor8. The first number of frames C1 indicates the number of other frameshaving regions which correspond to the moving body X0 and do not overlapwith the region corresponding to the moving body X0 in the referenceframe Fr4. The second number of frames C2 indicates the number of otherframes having regions which correspond to the moving body X0 and overlapwith the region corresponding to the moving body X0 in the referenceframe Fr4. The third number of frames C3 indicates the number of otherframes having regions which correspond to the moving body X0 and overlapwith each other. Note that the monitor 8 may display not only thecumulative frame Frc, but also an image obtained by overlapping theframes Fr1 to Fr4.

[0143] Herein, in the first embodiment, a composite image G0 is obtainedfrom four frames Fr1 to Fr4. The first number-of frames C1 indicatingthe number of other frames having regions, which correspond to themoving body X0 and do not overlap with the region corresponding to themoving body X0 in the reference frame Fr4, is 2. The second number offrames C2 indicating the number of other frames having regions, whichcorrespond to the moving body X0 and overlap with the regioncorresponding to the moving body X0 in the reference frame Fr4, is 1.The third number of frames C3 indicating the number of other frameshaving regions, which correspond to the moving body X0 and overlap witheach other, is 0. Hence, the condition C1>C2>C3 is 2>1>0, and thussatisfied. Therefore, in the second embodiment, when the operator inputsC1=2, C2=1, and C3=0 from the inputting means 6, the judging means 7judges that the condition C1>C2>C3 is satisfied and allows the regioncutout means 4 to cut out the moving body regions.

[0144] Meanwhile, FIG. 9 shows an example of a cumulative frame Frcobtained from six frames Fr1 to Fr6. In the cumulative frame Frc shownin FIG. 9, a reference frame is the frame Fr6, which is most recent inchronological order, and the moving body X0 moves from the left to rightin the drawing. Numbers in the cumulative frame Frc indicate cumulativevalues. In the example shown in FIG. 9, the first number of frames C1 is3. The second number of frames C2 is 2. The third number of frames C3is 1. Accordingly, the condition C1>C2>C3 is satisfied. Thus, when theoperator inputs the values of C1=3, C2=2, C3=1 from the inputting means6, the judging means 7 judges that the condition C1>C2>C3 is satisfiedand allows the region cutout means 4 to cut out the moving body regions.

[0145]FIG. 10A shows an example of a cumulative frame Frc obtained fromthree frames Fr1 to Fr3. FIG. 10B shows an example of a cumulative frameFrc obtained from two frames Fr1 and Fr2. In the cumulative frames Frcshown in FIGS. 10A and 10B, the reference frames are Fr3 and Fr2, whichare most recent in chronological order, and the moving body X0 is movingfrom the left to right in the drawing.

[0146] In the example shown in FIG. 10A, the first number of frames C1is 1. The second number of frames C2 is 1. The third number of frames C3is 1. Accordingly, the condition C1>C2>C3 is not satisfied. Thus, whenthe operator inputs the values of C1=1, C2=1, and C3=1 from theinputting means 6, the judging means 7 judges that the conditionC1>C2>C3 is not satisfied and does not allow the region cutout means 4to cut out the moving body regions. In the example shown in FIG. 10B,the first number of frames C1 is 0. The second number of frames C2 is 1.The third number of frames is 0. Accordingly, the condition C1>C2>C3 isnot satisfied. Thus, when the operator inputs the values of C1=0, C2=1,and C3=0 from the inputting means 6, the judging means 7 judges that thecondition C1>C2>C3 is not satisfied and does not allow the region cutoutmeans 4 to cut out the moving body regions.

[0147] As described above, in the second embodiment, the inputting means6 accepts the inputs of the first to third numbers of frames C1 to C3,and the judging means 7 judges whether the condition C1>C2>C3 issatisfied. Only when the condition C1>C2>C3 is satisfied, the judgingmeans 7 allows the region cutout means 4 to cut out the moving bodyregions. Therefore, it is possible to ensure that the moving body regioncorresponding to the moving body X0 is cut out from each frame.

[0148] Next, a third embodiment of the present invention is described.FIG. 11 is a schematic block diagram showing the configuration of asystem for composing a moving image according to the third embodiment ofthe present invention. Note that the components in the third embodiment,which are the same as those in the first embodiment, are given the samereference numerals, and details thereof are omitted.

[0149] The system for composing a moving image according to the thirdembodiment further comprises selecting means 9, in addition to thecomponents in the system for composing a moving image according to thefirst embodiment. The selecting means 9 compares the sizes of maskregions in each of mask frames with a third threshold Th3 and selects amask frame, of which the size of the mask region exceeds the thirdthreshold Th3. Second mask frame calculating means 3 employs only themask frame selected by the selecting means 9 to calculate a referencemask frame Frc1.

[0150] Herein, assume a case in which a composite image G0 is obtainedby using five frames Fr1 to Fr5 shown in FIG. 12. When the frame Fr5 isset to be the reference frame, mask frames M1 to M4 corresponding to theframes Fr1 to Fr4 are as shown in FIG. 13. Meanwhile, a cumulative frameFrc is as shown in FIG. 14A. The reference mask frame Frc1, which isobtained by binarizing the cumulative frame Frc based on the thresholdTh2 (Th2 is 2), is as shown in FIG. 14B.

[0151] The calculation of the composite image G0 based on these obtainedmask frames M1 to M4 and the reference mask frame Frc1 is described withreference to FIG. 15. A first composite image GFr1 is obtained byoverwriting moving body regions A1A and A1B, which are cut out from theframe Fr1, on the frame Fr1. Note that the moving body region A1Acorresponds to a region of a moving body X0 in the frame Fr1, and themoving body region A1B corresponds to a region of the moving body X0 inthe frame Fr5.

[0152] Next, a second composite image GFr1+2 is obtained by overwritingmoving body regions A2A and A2B, which are cut out from the frame Fr2,on the first composite image GFr1. Note that the moving body region A2Acorresponds to a region of the moving body X0 in the frame Fr2, and themoving body region A2B corresponds to the region of the moving body X0in the frame Fr5. Consequently, the moving body region A2B isoverwritten on the moving body region A1B in the composite image GFr1.

[0153] Subsequently, a third composite image GFr1+2+3 is obtained byoverwriting moving body regions A3A and A3B, which are cut out from theframe Fr3, on the second composite image GFr1+2. Note that the movingbody region A3A corresponds to a region of the moving body X0 in theframe Fr3, and the moving body region A3B corresponds to the region ofthe moving body X0 in the frame Fr5. Accordingly, the moving body regionA3B is overwritten on the moving body region A2B partially in thecomposite image GFr1+2.

[0154] Moreover, a fourth composite image GFr1+2+3+4 is obtained byoverwriting moving body regions A4A and A4B, which are cut out from theframe Fr4, on the third composite image GFr1+2+3. Note that the movingbody region A4A corresponds to part of a region of the moving body X0 inthe frame Fr4, and the moving body region A4B corresponds to part of aregion of the moving body X0 in the frame Fr5. Accordingly, the movingbody region A4B is overwritten on the moving body region A3B in thecomposite image GFr1+2+3.

[0155] At last, a fifth composite image GFr1+2+3+4+5, in other words, afinal composite image G0 is obtained by overwriting a moving body regionA5A, which is cut out from the frame Fr5, on the fourth composite imageGFr1+2+3+4.

[0156] Herein, part of the circular moving body X0 is missing in thereference mask frame Frc1. Thus, a region K0 without the moving body X0appears in a portion in the obtained composite image G0, where themoving body regions A3A, A4A, and A5A are overlapped.

[0157] As a result, in the third embodiment, the selecting means 9compares the sizes of the mask regions in the mask frames M1 to M4 withthe threshold Th3. Thereafter, the selecting means 9 removes the maskframes, of which size of the mask region is the threshold Th3 or less,from the calculation of the reference mask frame Frc1. Accordingly, theselecting means 9 selects only the mask frames, of which size of themask region exceeds the threshold Th3, for the calculation of thereference mask frame Frc1.

[0158] Herein, the mask region of the mask frame M4 is the smallestamong the mask regions of the mask frames M1 to M4 shown in FIG. 13, interms of size. Hence, by setting the threshold Th3 to an appropriatevalue (e.g., a half of the maximum value of an area among the individualmask regions in the mask frames M1 to M4), the selecting means 9 selectsthe mask frames M1 to M3 for the calculation of the reference mask frameFrc1.

[0159] The second mask frame calculating means 3 calculates thecumulative frame Frc by accumulating the selected mask frames M1 to M3.FIG. 16A shows the cumulative frame Frc obtained from the mask frames M1to M3. FIG. 16B shows the reference mask frame Frc1 obtained bybinarizing the cumulative frame Frc shown in FIG. 16A based on thethreshold Th2 (Th2 is 1).

[0160] When the moving body region A5A is cut out from the frame Fr5based on the calculated reference mask frame Frc1, it is possible to cutout the moving body region A5A without missing part of the moving bodyX0 included in the frame Fr5. Accordingly, the fifth composite imageGFr1+2+3+4+5, that is, the final composite image G0, which is obtainedby overwriting the moving body region A5A cut out from the frame Fr5 onthe fourth composite image GFr1+2+3+4, has a complete moving body X0 inthe final frame Fr5 as shown in FIG. 17. Therefore, it is possible toobtain the composite image G0 including the movement trajectory of themoving body X0 without missing part of the moving body X0.

[0161] Note that the system for composing a moving image in the thirdembodiment may further comprise inputting means 6, judging means 7, anda monitor 8 as in the second embodiment. Accordingly, the system mayaccept the inputs of the first number of frames C1, the second number offrames C2, and the third number of frames C3 and cut out the moving bodyregions from the frames by the region cutout means 4 only when thecondition C1>C2>C3 is satisfied.

[0162] Next, a fourth embodiment of the present invention is described.FIG. 18 is a schematic block diagram showing the configuration of asystem for composing a moving image according to a fourth embodiment ofthe present invention, which includes a device for cutting out a movingimage. Note that the components in the fourth embodiment, which are thesame as those in the first embodiment, are given the same referencenumerals, and details thereof are omitted.

[0163] The system for composing a moving image according to the fourthembodiment further comprises color information calculating means 20, inaddition to the components in the system for composing a moving imageaccording to the first embodiment. The color information calculatingmeans 20 calculates color information on a region on a reference frame,which corresponds to a mask region of a reference mask frame Frc1. Thecolor information calculating means then weights a region on acumulative frame Frc, which corresponds to a region having colorinformation similar to the calculated color information. The weightedcumulative frame Frc is binarized based on a threshold Th2 tore-calculate the reference mask frame Frc1.

[0164] Herein, assume a case in which a composite image G0 is obtainedby using five frames Fr1 to Fr5 as shown in FIG. 12, similar to thethird embodiment. When the frame Fr5 is set to be the reference frame,mask frames M1 to M4 corresponding to the frames Fr1 to Fr4 are as shownin FIG. 13. Meanwhile, a cumulative frame Frc is as shown in FIG. 14A.The reference mask frame Frc1, which is obtained by binarizing thecumulative frame Frc based on the threshold Th2 (Th2 is 2), is as shownin FIG. 14B.

[0165] Consequently, part of the circular moving body X0 is missing inthe reference mask frame Frc1. Thus, as previously mentioned, the regionK0 without the moving body X0 appears in a portion, where the movingbody regions A3A, A4A, and A5A are overlapped, in the composite image G0obtained by using the reference mask frame Frc1.

[0166] Therefore, with the premise that the moving body X0 hassubstantially the same color, the color information calculating means 20in the fourth embodiment cuts out the moving body region A5A from thereference frame Fr5 by using the previously calculated reference maskframe Frc1, calculates the color information on the moving body regionA5A, and re-calculates the reference mask frame Frc1 by using the colorinformation.

[0167] Specifically, the color information calculating means 20 obtainsa histogram of the cutout moving body region A5A and calculates a colorrepresented by a median value of the histogram as the color informationon the moving body region A5A. Note that color data RGB or luminance andchrominance data YCC form the moving image data D0 and the frames Fr1 toFr5. Accordingly, the median value of the histogram is calculated usingeach component of RGB or the chrominance components of YCC. A colorformed of the median value of each component is calculated as the colorinformation. Note that a color having a mean value of the histogram or afrequency more than or equal to a predetermined value in the histogrammay be calculated, instead of having the median value of the histogram.

[0168] Next, the color information calculating means 20 obtains a colorregion having a color similar to that represented by the calculatedcolor information on the reference frame Fr5. Specifically, a region,which has a color within a predetermined color space range with a centerof the color information calculated in the color space, is obtained asthe color region. Herein, part of the circular moving body X0 is missingon the moving body region A5A shown in FIG. 14B, which is cut out fromthe reference frame Fr5 by using the reference mask frame Frc1. However,since the color information on the moving body region A5A represents acolor of the moving body X0, the color region obtained by the calculatedcolor information corresponds to the circular object X0 as shown in FIG.19.

[0169] Subsequently, the color information calculating means 20 weightsthe cumulative frame Frc by adding the cumulative value to the regioncorresponding to the color region on the cumulative frame Frc. Note thatthe cumulative value of 1 is added herein. FIG. 20A shows the cumulativeframe Frc to which the cumulative value is added. FIG. 20B shows thereference mask frame Frc1 obtained by binarizing the cumulative frameFrc shown in FIG. 20A with a threshold Th2 (Th2 is 2).

[0170] When the moving body region A5A is cut out from the frame Fr5 byusing the re-calculated reference mask frame Frc1, it is possible to cutout the moving body region A5A without missing part of the moving bodyX0 included in the frame Fr5. Accordingly, the fifth composite imageGFr1+2+3+4+5, that is, the final composite image G0, which is obtainedby overwriting the moving body region A5A cut out from the frame Fr5 onthe fourth composite image GFr1+2+3+4, has a complete moving body X0 inthe final frame Fr5 as shown in FIG. 17. Therefore, it is possible toobtain the composite image G0 including the movement trajectory of themoving body X0 without missing part of the moving body X0.

[0171] Note that the system for composing a moving image in the fourthembodiment may further comprise inputting means 6, judging means 7, anda monitor 8 as in the second embodiment. Accordingly, the system mayaccept the inputs of the first number of frames C1, the second number offrames C2, and the third number of frames C3 and cut out the moving bodyregions from the frames by the region cutout means 4 only when thecondition C1>C2>C3 is satisfied.

[0172] Next, a fifth embodiment of the present invention is described.FIG. 21 is a schematic block diagram showing the configuration of asystem for composing a moving image according to a fifth embodiment ofthe present invention, which includes a device for cutting out a movingimage. Note that the components in the fifth embodiment, which are thesame as those in the first embodiment, are given the same referencenumerals, and details thereof are omitted.

[0173] The system for composing a moving image according to the fifthembodiment further comprises aligning means 10 and frame correctingmeans 11, in addition to the components in the system for composing amoving image according to the first embodiment. Furthermore, first maskframe calculating means 2 further comprises filtering means 2C. Thealigning means 10 aligns a plurality of frames Fri sampled from thesampling means 1. The frame correcting means 11 corrects the pixelvalues of the plurality of aligned frames Fri. The filtering means 2Cfilters an absolute value |di(x, y)| of a difference value calculated bysubtracting means 2A.

[0174] Similar to the first embodiment, four frames Fr1 to Fr4 aresampled in the description below. However, the present invention is notlimited to this.

[0175] The aligning means 10 aligns other frames Fr1 to Fr3 with areference frame Fr4. FIG. 22 is a diagram for explaining the alignmentby the aligning means 10. Herein, the alignment of the frames Fr1 andFr4 is described. As shown in FIG. 22, each of the frames Fr1 and Fr4includes a circular moving body X0 and a cuboid background B0. Uponfilming, the background B0 is not supposed to be moving. However, notonly the moving body X0 but also the background B0 are moved between theframes Fr1 and Fr4 in actual cases due to wobbling or the like.Nevertheless, the movement amount of the background B0 differs from thatof the moving body X0. Meanwhile, the moving body X0 stays still whenthe movement amount of the moving body X0 agrees with that of thebackground B0. Note that FIG. 22 shows the movement amount of thebackground B0 in a larger scale to facilitate explanation.

[0176] The aligning means 10 moves the frame Fr1 parallel to the frameFr4. Thereafter, the aligning means 10 calculates the movement amountand direction of the frame Fr4, where the correlation becomes maximumbetween pixel values of each pixel Fr1 (x, y) of the frame Fr1 and eachpixel Fr4 (x, y) of the frame Fr4, as a motion vector V1 regarding theframe Fr1. The aligning means 10 moves the frame Fr1 parallel to themovement direction reverse to the motion vector V1, in other words, adirection of a motion vector −V1 to align the frame Fr1 with the frameFr4. Herein, since a region of the moving body X0 is smaller than thebackground B0, the background B0 affects the correlation value, ratherthan the moving body X0. Thus, the aligning means 10 aligns onlybackground regions between the frames Fr1 and Fr4.

[0177] Note that the aligning means 10 judges that the correlation ismaximal when a cumulative sum obtained by squaring the difference of thepixel values between the frames Fr1 and Fr4 or a cumulative sum of theabsolute values is minimal.

[0178] Herein, the motion vector V1 is calculated to move the frame Fr1parallel to align the frames Fr1 and Fr4. However, the frames Fr1 andFr4 may be aligned by using other methods such as affine transformationand the method disclosed in Japanese Patent Application No. 2002-249212.In the method disclosed in Japanese Patent Application No. 2002-249212,a reference patch including one or a plurality of rectangular region isdisposed on the reference frame (i.e., frame Fr4). A patch having thesame structure as the reference patch is disposed on the other frame(i.e., frame Fr1). The patch is moved and/or transformed on the frameFr1 so that an image in the patch may match an image in the referencepatch. Based on the reference patch and the moved and/or transformedpatch, a correspondence is estimated between a pixel in the patch on theframe Fr1 and a pixel in the reference patch on the frame Fr4. Theframes are aligned based on the correspondence.

[0179] When the frames Fr1 to Fr4 are formed of luminance andchrominance components YCC, the alignment may be performed in terms ofthe Y component, instead of all the components. As for the othercomponents, the alignment is performed similarly to the Y component.When the frames Fr1 to Fr4 are formed of RGB components, the alignmentmay be performed for only one of the RGB components, instead of all thecomponents. As for the other components, the alignment may be performedsimilarly to the component with which the alignment was performed.

[0180] By the alignment, a region where a pixel does not have a signalvalue (e.g., a shaded area in the aligned frame Fr1 in FIG. 22) appearsin the aligned frames Fr1 to Fr3. Thus, the frame correcting means 11,first mask frame calculating means 2, and second mask frame calculatingmeans 3, which are steps after the aligning means 10, correct theframes, calculate mask frames, and calculate a cumulative binary maskframe for only the region where the pixel has a signal value,respectively.

[0181] The frame correcting means 11 calculates a mean value of pixelvalues, which indicates a level of the pixel values for each of theframes Fr1 to Fr3 and the reference frame Fr4 before or after thealignment. The frame correcting means 11 corrects the pixel values ofthe plurality of frames Fr1 to Fr4 so that the mean values maysubstantially match.

[0182] Specifically, mean values FrM1 to FrM4 of all the pixel valuesare calculated for the frames Fr1 to Fr3 (after aligned) and thereference frame Fr4 by using the following equation (4):

FrMi=ΣFri(x, y)/(m×n)   (4)

[0183] where m×n is the total number of pixels having pixel values inthe frame Fri before or after the alignment.

[0184] As for the frames Fr1 to Fr3, the pixel values of the frames Fr1to Fr3 are corrected-by using the equation (5) below so that the meanvalue FrMi of all the pixel values may match the mean value FrM4 of allthe pixel values of the reference frame.

FriH(x,y)=Fri(x,y)+(FrM 4(x,y)−FrMi(x,y))   (5)

[0185] where FriH(x,y) is a pixel value of the corrected frame, and i is1, 2, or 3.

[0186] When the frames Fr1 to Fr4 are formed of luminance andchrominance components YCC, the frames Fr1 to Fr4 may be corrected interms of only the Y component, instead of all the components.

[0187] Instead of a mean value, any values that represent all the pixelsin the frames Fr1 to fr3, such as a median value of all the pixels and amaximum frequency value of a histogram, may be used.

[0188] The filtering means 2C filters the absolute values of thedifference values between the corresponding pixels in the referenceframe Fr4 and the other frames Fr1 to Fr3 (which are aligned and ofwhich pixel values are corrected), which are calculated by thesubtracting means 2A. FIG. 23 is a diagram showing an example of afilter used by the filtering means 2C. A filter F1 shown in FIG. 23 is alow pass filter which sets a mean value of pixel values of eightadjacent pixels including self as a self pixel value.

[0189] Herein, the regions where the absolute values are relativelylarge correspond originally to the moving body X0 included in thereference frame Fr4 and the other frames Fr1 to Fr3. However, each ofthe frames Fr1 to Fr4 includes noise upon filming and noise due to themovement of the moving body X0. Accordingly, the regions where theabsolute values are relatively large possibly include small regionswhere the absolute values are small. On the contrary, the regions wherethe absolute values are small possibly include small regions where theabsolute values are relatively large. Consequently, the filter F1 shownin FIG. 23 filters the absolute values of the difference values, andthus it is possible to remove the noise due to the small regions fromthe absolute values. Therefore, the mask frames M1 to M3 can beprecisely calculated while the effects of the noise are reduced.

[0190] Herein, a 3×3 low pass filter is used. However, a 5×5 or largerlow pass filter may be used. Alternatively, a median filter, a maximumvalue filter, or a minimum value filter may be used.

[0191] Next, the process performed in the fifth embodiment is described.FIG. 24 is a flow chart showing the process performed in the fifthembodiment. Herein, similar to the first embodiment, the four frames Fr1to Fr4 are sampled in the description.

[0192] First, sampling means 1 samples the four frames Fr1 to Fr4 frommoving image data D0 (Step S11). Second, the aligning means 10 alignsthe frames Fr1 to Fr3 with the reference frame Fr4 (Step S12). Third,the frame correcting means 11 corrects the pixel values of the aligned(or pre-aligned) frames Fr1 to Fr4 (Step S13).

[0193] Fourth, subtracting means 2A of the first mask frame calculatingmeans 2 calculates the absolute values |di(x, y)| of the differencevalues between the corresponding pixels in the reference frame Fr4 andthe other frames Fr1 to Fr3, among the plurality of aligned andcorrected frames Fr1 to Fr4 (Step S14). Fifth, the filtering means 2Cfilters the absolute values of the difference values by using the filterF1 shown in FIG. 23 (Step S15). Sixth, binarizing means 2B binarizes thefiltered absolute values of the difference values based on the thresholdTh1 to calculate the mask frames M1 to M3 (Step S16).

[0194] Seventh, accumulating means 3A of second mask frame calculatingmeans 3 accumulates the corresponding pixels for all the mask frames M1to M3 to calculate a cumulative frame Frc (Step S17). Eighth, binarizingmeans 3B binarizes the cumulative frame Frc based on the threshold Th2to calculate the reference mask frame Frc1 (Step S18).

[0195] Ninth, region cutout means 4 masks the frames Fr1 to Fr3 by usingthe corresponding mask frames M1 to M3. The region cutout means 4 alsomasks the reference frame Fr4 by using the reference mask frame Frc1.Accordingly, the region cutout means 4 cuts out moving body regions A1to A4, which correspond to the moving body X0, from the plurality offrames Fr1 to Fr4 (Step S19).

[0196] At last, composing means 5 overwrites the moving body regions A1to A4 in chronological order at positions in one of the frames Fr1 toFr4 (Fr1), which correspond to the moving body regions A1 to A4 toobtain a composite image G0 of the plurality of frames Fr1 to Fr4 (StepS20). The process is completed.

[0197] As described above, in the fifth embodiment, the aligning means10 aligns the background regions in the plurality of frames Fr1 to Fr4.Accordingly, the effects of the movement of the entire frame due towobbling or the like upon filming can be removed from the frames Fr1 toFr4. Therefore, it is possible to cut out the moving body regionsaccurately from the frames while the effects of the movement of theentire frame are reduced.

[0198] Herein, when the contrast adjusting function of a video cameraused for filming the moving image does not operate normally or when itbecomes suddenly cloudy during outdoor filming in fine weather, theentire contrast differs between the sampled frames. Thus, when the firstmask frame calculating means 2 calculates the absolute values |di(x, y)|of the difference values between the corresponding values in thereference frame Fr4 and the other frames Fr1 to Fr3 by using suchframes, the absolute values |di(x, y)| will have relatively large valueseven in the background regions which do not correspond to the movingbody X0. As a result, it is impossible to accurately calculate the maskframes M1 to M3 having the mask regions which correspond to only themoving body X0.

[0199] Thus, the frame correcting means 11 corrects the pixel values ofthe frames Fr1 to Fr4, and the entire contrast in each frame can besubstantially the same. Hence, the background regions, which do notcorrespond to the moving body X, will not have extremely large absolutevalues |di(x, y)|. As a consequence, it is possible to accuratelycalculate the mask frames M1 to M3 having mask regions which correspondto only the moving body X0.

[0200] The filtering means 2C of the first mask frame calculating means2 filters the absolute values |di(x, y)| of the difference values toremove the noise. Thus, it is possible to remove the noise from theabsolute values |di(x, y)|. Therefore, the mask frames M1 to M3 can beprecisely calculated while the effects of the noise are reduced.

[0201] Note that, in the fifth embodiment, the system may furthercomprise inputting means 6, judging means 7, and a monitor 8 as a sixthembodiment shown in FIG. 25.

[0202] In addition, in the fifth embodiment, filtering means 3C may beprovided in the second mask frame calculating means 3 as a seventhembodiment shown in FIG. 26. The filtering means 3C filters thecumulative frame Frc, which is calculated by the accumulating means 3A,by using the filter F1 shown in FIG. 23.

[0203] Consequently, it is possible to remove noise upon filming andnoise due to the movement of the moving body X0 from the cumulativeframe Frc. Therefore, the reference mask frame Frc1 can be preciselycalculated while the effects of the noise are reduced.

[0204] Furthermore, the aligning means 10 calculates the motion vectorV1 between the frames to align the frames in the fifth embodiment.However, the motion vector may be calculated in terms of only thebackground B0 by separating the moving body X0 and the background B0included in the frames. Hereinafter, a process in the aligning means 10for performing the alignment by separating the moving body X0 and thebackground B0 in the frames is described as an eighth embodiment. Notethat the alignment between the frames Fr1 and Fr4 as shown in FIG. 22 isdescribed herein.

[0205] First, the aligning means 10 obtains a difference frame Frs bycalculating absolute values of difference values in the correspondingpixels between the frames Fr1 and Fr4. The calculation of the absolutevalues of the difference values is performed using the aforementionedequation (1).

[0206]FIG. 27A is a diagram showing the difference frame Frs. Asindicated by shaded areas in FIG. 27A, moved portions between the framesFr1 and Fr4 have large values in the difference frame Frs. Specifically,portions indicating the movement amounts of a region corresponding tothe moving body X and a region corresponding to the cuboid background B0have large values.

[0207] Second, the aligning means 10 performs morphology on thedifference frame Frs. Hereinafter, the morphology is described.

[0208] The morphology has been studied as a technique effective fordetecting microcalcification, which is a pathognomonic form of breastcancer in particular. The object image is not limited to themicrocalcification in the mammogram. The morphology can be applied toany images in which the size or the shape of a particular image portion(abnormality, noise, or the like) is predefined to some extent (e.g.,U.S. Pat. Nos. 6,233,362 and 5,937,111 and Japanese Unexamined patentPublication No. 9(1997)-91421). The morphology is simply described belowwith an example of a grayscale image.

[0209] The morphology is a process to manipulate an object image byemploying a morphology element. The morphology element is relative to anorigin having a predetermined offset value and has a predetermined size.The morphology includes dilation, erosion, opening, and closing.

[0210] The dilation is a process to find the maximum value in a range ofa center pixel ±m (which is a value determined in accordance with themorphology element and corresponds to the size of the morphologyelement). The dilation expands bright regions and contracts dark regionsin the image.

[0211] The erosion is a process to find the minimum value in a range ofa center pixel ±m. The erosion contracts the bright regions and expandsthe dark region in the image.

[0212] The opening is a process to perform the dilation after theerosion, in other words, to find the maximum value after searching theminimum value. The opening removes redundant bright fine portions, finelines, and infinitesimal regions from the image.

[0213] The closing is a process to perform the erosion after thedilation, in other words, to find the minimum value after searching themaximum value. The closing removes redundant dark fine portions, finelines, and infinitesimal regions from the image.

[0214] In the eighth embodiment, bright fine portions, fine lines, andinfinitesimal regions are removed from the difference frame Frs byperforming the opening on the difference frame Frs. Herein, the movementamount of the moving body X0 included in the frames Fr1 and Fr4 islarge. Accordingly, as shown in FIG. 27A, it is possible to confirm thecircular shape in the difference frame Frs. However, since the movementamount of the cuboid background B0 is small, it is impossible to confirmthe shape in the difference frame Frs, and only a narrow region B1exists along the outline of the cuboid. Thus, by performing the openingon the difference frame Frs, the narrow region B1 included in thedifference frame Frs is removed from a difference frame Frs0 after theopening as shown in FIG. 27B. Therefore, only regions X1 and X2corresponding to the moving body X0 exist.

[0215] Third, as shown in FIG. 28, the aligning means 10 removes theregions X1 and X2, which correspond to the moving body X0 in thedifference frame Frs0 after the process, from the frames Fr1 and Fr4.The aligning means 10 moves the frame Fr1 parallel to the frame Fr4after the regions X1 and X2 are removed and calculates the motion vectorV1 regarding the frame Fr1 as previously mentioned. The aligning means10 moves the frame Fr1 parallel to the motion vector −V1 and aligns theframe Fr1 with the frame Fr4.

[0216] As described above, the regions X1 and X2 corresponding to themoving body X0 are removed from the frames Fr1 and Fr4, and the framesFr1 and Fr4 are aligned. Thus, it is possible to obtain the motionvector V1 precisely by eliminating the effects of the moving body X0having larger movement than the background B0. Therefore, the frames Fr1and Fr4 can be aligned precisely.

[0217] Note that the subtracting means 2A calculates the absolute valuesof the difference values between each frame and the reference frame tocalculate the mask frames using frames except for the reference frame inthe aforementioned embodiments. However, the mask frames may becalculated by calculating the absolute values of the difference valuesbetween the frames chronologically adjacent to each other. This isdescribed as a ninth embodiment below.

[0218]FIG. 29 is a diagram for explaining the process performed in theninth embodiment by using the frames. Similar to the first embodiment,the process using four frames Fr1 to Fr4 is described in the ninthembodiment. In the ninth embodiment, as shown in FIG. 29, mask framesM1′ to M3′ are calculated by calculating and binarizing absolute valuesof difference values between the frames Fr1 and Fr2, absolute values ofdifference values between the frames Fr2 and Fr3, and absolute values ofdifference values between the frames Fr3 and Fr4. The mask frame M1′ hasmask regions R1A′ and R1B′. The mask frame M2′ has mask regions R2A′ andR2B′. The mask frame M3′ has mask regions R3A′ and R3B′. Herein, themask regions R1A′, R2A′, and R3A′ of the mask frames M1′ to M3′correspond to regions of a moving body X0 in the frames Fr1 to Fr3,respectively. The mask regions R1B′, R2B′, and R3B correspond to regionsof the moving body X0 in the adjacent frames.

[0219] The frames Fr1 to Fr3 are masked by using the corresponding maskframes M1′ to M3′, and the reference frame Fr4 is masked by using thereference mask frame Frc1. Accordingly, a moving body regioncorresponding to the moving body X0 are cut out from each of theplurality of frames Fr1 to Fr4. Specifically, as for the frame Fr1,regions, which correspond to the mask regions R1A′ and R1B′ having thepixel values of 1 in the mask frame M1′, are cut out as moving bodyregions A1A′ and A1B′ (hereinafter, generically referred to as A1′occasionally). As for the frame Fr2, regions, which correspond to themask regions R2A′ and R2B′ having the values of 1 in the mask frame M2′,are cut out as moving body regions A2A′ and A2B′ (hereinafter,generically referred to as A2′ occasionally). As for the frame Fr3,regions, which correspond to the mask regions R3A′ and R3B′ having thevalues of 1 in the mask frame M3′, are cut out as moving body regionsA3A′ and A3B′ (hereinafter, generically referred to as A3′occasionally).

[0220] Meanwhile, as for the reference frame Fr4, a region, whichcorresponds to the mask region R4A′ having the value of 1 in thereference mask frame Frc1, is cut out as a moving body region A4A′(hereinafter, referred to as A4′ occasionally).

[0221] Similar to the first embodiment, a cumulative frame Frc of themask frames M1 to M3, which are calculated based on the absolute valuesof the difference values between each frame and the reference frame, iscalculated. Then, the cumulative frame Frc is binarized to calculate thereference mask frame Frc1.

[0222] The moving body regions A1′ to A4′ are overwritten inchronological order at the positions corresponding to the moving bodyregions A1′ to A4′ in one of the frames Fr1 to Fr4 (herein, Fr1). Thus,a composite image G0′ of the plurality of frames Fr1 to Fr4 is obtained.

[0223] Note that, in the fifth embodiment, the frame correcting means 11calculates a mean value of pixel values, which indicates a level of thepixel values for each of the plurality of frames Fr1 to Fr4. The framecorrecting means 11 corrects the pixel values of the plurality of framesFr1 to Fr4 so that the mean values may substantially match. Accordingly,the entire brightness is substantially the same in each of the frames,and background regions which do not correspond to the moving body X0will not have extremely large absolute values |di(x, y)|. Thus, the maskframes M1 to M3 having mask regions corresponding to only the movingbody X0 are appropriately calculated. However, the same effects can beobtained by correcting the threshold Th1 used for the binarization bythe binarizing means 2B of the first mask frame calculating means 2.Hereinafter, the correction of the threshold Th1 is described as a tenthembodiment.

[0224]FIG. 30 is a schematic block diagram showing the configuration ofa system for composing a moving image according to the tenth embodimentof the present invention, which includes a device for cutting out amoving image. Note that the components in the tenth embodiment, whichare the same as those in the first embodiment, are given the samereference numerals, and details thereof are omitted.

[0225] The system for composing a moving image according to the tenthembodiment further comprises threshold correcting means 2D, in additionto the components in the first embodiment. The threshold correctingmeans 2D corrects a threshold Th1 for each absolute value |di(x, y)| ofdifference values. The threshold Th1 is used for the binarization by thebinarizing means 2B of the first mask frame calculating means 2.Hereinafter, the correction of the threshold Th1 is described. Note thatfour frames Fr1 to Fr4 are also sampled from moving image data D0 in thetenth embodiment.

[0226] First, the threshold correcting means 2D calculates mean valuesFrM1 to FrM4 of all pixel values in the frames Fr1 to Fr3 and referenceframe Fr4 by using the aforementioned equation (4). Absolute values ofdifference values between the mean value FrM4 of the reference frame Fr4and the mean values FrM1 to FrM3 of the other frames Fr1 to Fr3 arecalculated as correction coefficients αi (i=1 to 3) using the equation(6) below:

αi=|FrM 4−FrMi|  (6).

[0227] Herein, the binarizing means 2B binarizes the absolute values|di(x, y)| of the difference values calculated by using theaforementioned equations (1) and (2) based on the threshold Th1.However, in the tenth embodiment, the threshold Th1 is corrected foreach of the absolute values |di(x, y)| of the difference values toobtain a corrected threshold Th1′i (i=1 to 3). The binarization isperformed based on the corrected threshold Th1′i. The correctedthreshold Th1′i is calculated by using the equation (7) below:

Th 1′i=Th 1+αi   (7).

[0228] The binarizing means 2B binarizes the absolute values |di(x, y)|of the difference values based on the corrected threshold Th1′i.Specifically, a value of 1 is given to a pixel having the absolute value|di(x, y)| of the difference value larger than the threshold Th1′i. Avalue of 0 is given to a pixel having the absolute value less than orequal to the threshold Th1′i. Mask frames M1 to M3, which include pixelswith a value of 1 as mask regions and correspond to the other frames Fr1to Fr3, are calculated.

[0229] More specifically, the absolute value |d1(x, y)| of thedifference value calculated from the frame Fr1 and the reference frameFr4 is binarized based on a corrected threshold Th1′1. The correctedthreshold Th1′1 is calculated by correcting the threshold Th1 using thecorrection coefficient α1 obtained from the mean values FrM4 and FrM1.The absolute value |d2(x, y)| of the difference value calculated fromthe frame Fr2 and the reference frame Fr4 is binarized based on acorrected threshold Th1′2. The corrected threshold Th1′2 is calculatedby correcting the threshold Th1 using the correction coefficient α2obtained from the mean values FrM4 and FrM2. The absolute value |d3(x,y)| of the difference value calculated from the frame Fr3 and thereference frame Fr4 is binarized based on a corrected threshold Th1′3.The corrected threshold Th1′3 is calculated by correcting the thresholdTh1 using the correction coefficient α3 obtained from the mean valuesFrM4 and FrM3.

[0230] Note that, in the tenth embodiment, the mean values of the framesFr1 to Fr4 are calculated to obtain the correction coefficient αi.However, instead of the mean values, any values representing all thepixels in the frames Fr1 to Fr4, such as median values of all the pixelsand the maximum frequency values of the histogram, may be used.

[0231] Moreover, in the tenth embodiment, when luminance and chrominancecomponents YCC form the frames Fr1 to Fr4, the threshold Th1 may becorrected by calculating the correction coefficient αi in terms of onlythe Y component, instead of all the components.

[0232] Note that the systems in the aforementioned first to thirdembodiments, fifth, sixth, eighth, and ninth embodiments may comprise atleast one means among the aligning means 10, the frame correcting means11, and the filtering means 2C.

[0233] In addition, the system in the tenth embodiment may compriseeither the aligning means 10 or the filtering means 2C.

What is claimed is:
 1. A device for cutting out a moving image,comprising: sampling means for sampling a plurality of sequential framesfrom the moving image including a moving body; mask frame calculatingmeans for calculating mask frames, which include mask regions foridentifying positions of the moving body in the plurality of frames,corresponding to each of the plurality of frames; and region cutoutmeans for cutting out moving body regions corresponding to the movingbody from all or part of the plurality of frames by masking all or partof the plurality of frames using the mask frames which correspondthereto.
 2. The device for cutting out a moving image according to claim1, wherein the mask frame calculating means is a means for calculatingother mask frames, which includes the mask regions for identifying thepositions of the moving body in other frames, corresponding to each ofthe other frames by calculating absolute values of difference values incorresponding pixels between a reference frame selected from theplurality of frames and the other frames and binarizing the absolutevalues based on a first threshold, and the region cutout means is ameans for cutting out the moving body regions corresponding to themoving body from the other frames by masking the other frames using theother mask frames which correspond thereto.
 3. The device for cuttingout a moving image according to claim 2, wherein the mask framecalculating means is a means for filtering the absolute values of thedifference values to remove noise and binarizing the absolute valueswhich have been filtered.
 4. The device for cutting out a moving imageaccording to claim 2, further comprising: threshold correcting means forcalculating a representative value of pixel values indicating a level ofthe pixel values in each of the plurality of frames to correct the firstthreshold for each of the absolute values of the difference values basedon the representative value, wherein the mask frame calculating means isa means for binarizing the absolute values based on the first thresholdcorrected for each of the absolute values of the difference values. 5.The device for cutting out a moving image according to claim 1, whereinthe mask frame calculating means comprises: first mask frame calculatingmeans for calculating other mask frames, which includes the mask regionsfor identifying the positions of the moving body in other frames,corresponding to each of the other frames by calculating absolute valuesof difference values in corresponding pixels between a reference frameselected from the plurality of frames and the other frames andbinarizing the absolute values based on a first threshold; and secondmask frame calculating means for calculating a reference mask frame,which includes a mask region for identifying a position of the movingbody in the reference frame, based on information on the plurality ofother mask frames, and the region cutout means is a means for cuttingout a moving body region corresponding to the moving body from thereference frame by masking the reference frame using the reference maskframe.
 6. The device for cutting out a moving image according to claim5, wherein the first mask frame calculating means is a means forfiltering the absolute values of the difference values to remove noiseand binarizing the absolute values which have been filtered.
 7. Thedevice for cutting out a moving image according to claim 5, furthercomprising: threshold correcting means for calculating s representativevalue of pixel values indicating a level of the pixel values in each ofthe plurality of frames to correct the first threshold for each of theabsolute values of the difference values based on the representativevalue, wherein the first mask frame calculating means is a means forbinarizing the absolute values based on the first threshold correctedfor each of the absolute values of the difference values.
 8. The devicefor cutting out a moving image according to claim 5, wherein the secondmask frame calculating means is a means for calculating the referencemask frame by accumulating or weighting and accumulating correspondingpixels in a selection mask frame selected from the plurality of othermask frames and further binarizing the accumulated selection mask framebased on a second threshold.
 9. The device for cutting out a movingimage according to claim 8, further comprising: selecting means forcomparing a size of the mask region in each of the other mask frameswith a third threshold to select the other mask frame with the maskregion having the size exceeding the third threshold as the selectionmask frame.
 10. The device for cutting out a moving image according toclaim 9, wherein the second mask frame calculating means is a means forfiltering the accumulated selection mask frame to remove noise andbinarizing the accumulated selection mask frame which has been filtered.11. The device for cutting out a moving image according to claim 9,wherein the second mask frame calculating means is a means forre-calculating the reference mask frame by calculating color informationon a region in the reference frame, which corresponds to the mask regionin the reference mask frame; weighting the region in the accumulatedselection mask frame, which corresponds to a region having colorinformation similar to the color information; and binarizing theweighted and accumulated selection mask frame based on the secondthreshold.
 12. The device for cutting out a moving image according toclaim 1, wherein the mask frame calculating means comprises: first maskframe calculating means for calculating other mask frames, whichincludes the mask regions for identifying the positions of the movingbody in other frames, corresponding to each of the other frames bycalculating absolute values of difference values in corresponding pixelsbetween a reference frame selected from the plurality of frames and theother frames and binarizing the absolute values based on a firstthreshold; and second mask frame calculating means for calculating areference mask frame, which includes a mask region for identifying aposition of the moving body in the reference frame, based on informationon the plurality of other mask frames, and the region cutout means isfor cutting out the moving body regions corresponding to the moving bodyfrom the plurality of frames by masking the other frames using the othermask frames which correspond thereto and masking the reference frameusing the reference mask frame.
 13. The device for cutting out a movingimage according to claim 12, wherein the first mask frame calculatingmeans is a means for filtering the absolute values of the differencevalues to remove noise and binarizing the absolute values which havebeen filtered.
 14. The device for cutting out a moving image accordingto claim 12, further comprising: threshold correcting means forcalculating s representative value of pixel values indicating a level ofthe pixel values in each of the plurality of frames to correct the firstthreshold for each of the absolute values of the difference values basedon the representative value, wherein the first mask frame calculatingmeans is a means for binarizing the absolute values based on the firstthreshold corrected for each of the absolute values of the differencevalues.
 15. The device for cutting out a moving image according to claim12, wherein the second mask frame calculating means is a means forcalculating the reference mask frame by accumulating or weighting andaccumulating corresponding pixels in a selection mask frame selectedfrom the plurality of other mask frames and further binarizing theaccumulated selection mask frame based on a second threshold.
 16. Thedevice for cutting out a moving image according to claim 15, furthercomprising: selecting means for comparing a size of the mask region ineach of the other mask frames with a third threshold to select the othermask frame with the mask region having the size exceeding the thirdthreshold as the selection mask frame.
 17. The device for cutting out amoving image according to claim 16, wherein the second mask framecalculating means is a means for filtering the accumulated selectionmask frame to remove noise and binarizing the accumulated selection maskframe which has been filtered.
 18. The device for cutting out a movingimage according to claim 16, wherein the second mask frame calculatingmeans is a means for re-calculating the reference mask frame bycalculating color information on a region in the reference frame, whichcorresponds to the mask region in the reference mask frame; weightingthe region in the accumulated selection mask frame, which corresponds toa region having color information similar to the color information; andbinarizing the weighted and accumulated selection mask frame based onthe second threshold.
 19. The device for cutting out a moving imageaccording to claim 2, further comprising: inputting means for acceptinginputs of a first number of frames, a second number of frames, and athird number of frames, the first number of frames indicating a numberof the other frames having a region which corresponds to the moving bodyand does not overlap with a region corresponding to the moving body inthe reference frame, the second number of frame indicating a number ofthe other frames having a region which corresponds to the moving bodyand overlaps with the region corresponding to the moving body in thereference frame, and the third number of frames indicating a number ofthe other frames having regions which correspond to the moving body andoverlap with each other, in a case where the plurality of frames areoverlapped; and judging means for judging whether a condition the firstnumber of frames>the second number of frames>the third number of framesexists among the first number of frames, the second number of frames,and the third number of frames, which are inputted, and for allowing theregion cutout means to cut out the moving body region only when thecondition is satisfied.
 20. The device for cutting out a moving imageaccording to claim 1, further comprising: aligning means for aligningbackground regions excluding the moving body regions in the plurality offrames, wherein the mask frame calculating means is a means forcalculating the mask frames from the plurality of frames which have beenaligned.
 21. The device for cutting out a moving image according toclaim 1, further comprising: frame correcting means for calculating arepresentative value of pixel values indicating a level of the pixelvalues in each of the plurality of frames to correct the pixel values ofthe plurality of frames so that the representative value may matchanother substantially, wherein the mask frame calculating means is ameans for calculating the mask frames from the plurality of frames whichhave been corrected.
 22. The device for cutting out a moving imageaccording to claim 1, further comprising: composing means for obtaininga composite image of the plurality of frames by overwriting the movingbody regions, which are cut out from all or part of the plurality offrames by the region cutout means, in chronological order at positionscorresponding to the moving body region in one of the plurality offrames.
 23. A method for cutting out a moving image, comprising thesteps of: sampling a plurality of sequential frames from the movingimage including a moving body; calculating mask frames, which includemask regions for identifying positions of the moving body in theplurality of frames, corresponding to each of the plurality of frames;and cutting out moving body regions corresponding to the moving bodyfrom all or part of the plurality of frames by masking all or part ofthe plurality of frames using the mask frames which correspond thereto.24. A program for causing a computer to execute a method for cutting outa moving image including procedures for: sampling a plurality ofsequential frames from the moving image including a moving body;calculating mask frames, which include mask regions for identifyingpositions of the moving body in the plurality of frames, correspondingto each of the plurality of frames; and cutting out moving body regionscorresponding to the moving body from all or part of the plurality offrames by masking all or part of the plurality of frames using the maskframes which correspond thereto.
 25. A computer readable medium havingrecorded therein a program for causing a computer to execute a methodfor cutting out a moving image including procedures for: sampling aplurality of sequential frames from the moving image including a movingbody; calculating mask frames, which include mask regions foridentifying positions of the moving body in the plurality of frames,corresponding to each of the plurality of frames; and cutting out movingbody regions corresponding to the moving body from all or part of theplurality of frames by masking all or part of the plurality of framesusing the mask frames which correspond thereto.